Natural language processing and dialogue systems, such as chatbots, are now becoming commonplace in many fields. Natural language processing relates to the methods by which computers process and analyse natural language data. Dialogue systems (or conversational agents) are computer systems intended to converse with humans in a coherent manner.
Newer approaches to dialogue systems powered by machine learning have garnered much interest and increasing success. Having said this, there are still a multitude of obstacles to their adoption including the difficulty in training and validating machine learning models. This is especially relevant in domains such as health care, where quality annotated datasets are limited and careful validation is crucial for safety and regulatory reasons.
Given these constraints, manual dialogue creation and management is still common in industry. Such dialogues are usually crafted by domain experts, who typically think of conversations as a form of tree with distinct choices corresponding to the various paths. A conversational agent operating under these conditions is referred to herein as a constrained-response chatbot as the valid user responses (e.g. answers) to dialogue prompts (e.g. questions) from the system are constrained. These are generally manifested in text with multiple choice questions or similar visual elements.
The downsides of constrained-response chatbots include a lack of flexibility, naturalness, and usability of the conversation flows. This is especially problematic in the case of voice support, where the inability to constrain user inputs often necessitates a rigid “phone tree”—like dialogue system (“Press 1 for . . . ”), or training a separate classification model specific to each branching step in order to send users down the correct path. Supporting voice modality forces a constrained-response chatbot to become a free-response chatbot, one that accepts any arbitrary utterance. Having said this, converting to free-response chatbots in a manner that is flexible and robust yet still accurate and safe is a great challenge.